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This  section  describes  the  scientific  accomplishments  for  the  project  in  relation  to  each  of  the  proposed  efforts  of  the  STIR 
proposal.  See  attachment  for  related  images. 

A.  Achievement  of  Aim  1:  Wireless,  Real-time  f-EEG  Experimentation  System 

Since  the  start  of  this  STIR  contract  (August  2014),  the  focus  of  the  effort  has  mainly  been  on  development,  refinement,  and 
integration  of  the  portable,  wireless,  real-time  f-EEG  system  proposed  in  Aim  #1.  The  system  uses  a  commercial  off-the-shelf 
(COTS)  16-node,  dry  electrodes  EEG  head  mounted  device  (Emotiv  Inc.,  Figure  1)  for  collection  of  EEG  data.  This  device  is 
wireless  and  transmits  data  via  Bluetooth  to  a  PC  using  a  USB  dongle.  The  advantage  of  the  system  over  others  is  the  ability  to 
collect  high  resolution  EEG  data  without  complicated  setup  and  calibration  as  with  EEG  nets  and  similar  devices.  For  the  end 
user,  donning  the  device  is  similar  to  wearing  and  positioning  of  headphones.  Further,  data  output  and  sensitivity  is  minimally 
affected  by  hair  and  other  artifact. 

The  EEG  headset  integrates  with  a  custom  designed  EEG  experimental  management,  data  collection  and  analysis  platform 
comprised  of  both  hardware  and  software  components.  At  the  heart  of  the  system  is  the  EEG  Experiment  Runtime  Service 
(eERS)  which  supports  experimental  set-up,  experimental  design  and  configuration,  set-up  for  incorporation  of  visual  stimuli, 
management  of  incoming  data,  and  data  analytics.  eERS  is  developed  with  an  open  source  strategy  and  future  deployment  is 
expected  to  be  cloud-based  to  allow  for  scalability,  portability  and  versatility.  Figure  2  shows  the  EEG  experimental  platform 
architecture. 

The  eERS  system  allows  for  single  or  multi-user  EEG  analyses.  For  multi-user  analyses,  each  user  focuses  on  his/her  own 
screen  (i.e.  viewpoint)  and  data  is  collected  using  the  personalized  headset.  This  EEG  data  is  managed  and  synchronized  with 
other  users,  as  appropriate,  through  the  collection  proxy  and  is  also  synchronized  with  events  in  visual  stimuli.  Care  in 
development  has  been  taken  to  address  issues  of  data  latency  resulting  from  incoming  transmission  from  remote  users. 
Additionally,  missing  packets  resulting  are  also  handled  by  the  collection  proxy.  Once  data  arrives,  it  is  processed  by  the  data 
analytics  engine  which  integrates  open-source  analytics  engine  by  Octave.  Algorithms  are  developed  for  key  EEG  feature 
extraction.  EEG  data  from  each  of  the  16  headset  nodes  is  combined  to  provide  3D  mapping  of  activated  brain  regions 
responding  to  stimuli.  Figure  3  provides  a  sample  of  raw  EEG  collected  from  the  wireless,  EEG  headset  system  along  with  a 
fused  mapping  in  3D  of  activated  regions  responding  to  stimuli. 

An  important  aspect  of  the  development  of  the  eERS  system  includes  the  development  of  the  experimental  set-up  engine 
(Figure  4).  This  portion  of  the  software  development  allows  for  customized  experimental  set-up  by  trained  as  well  as  minimally 
trained  users.  It  utilizes  an  icon-driven,  drag-n-drop  experimental  setup  schema  where  key  components  required  for  the 
experiment  (i.e.  single  vs.  multi  user,  visual  stimuli,  data  outputs,  etc)  can  be  easily  designed  and  implemented.  Various  stimuli 
(i.e.  images,  video,  audio)  as  well  as  input  devices  (i.e.  keyboard,  mouseclicks)  can  be  integrated  and  collected  by  the  eERS 
system. 

B.  Achievement  of  Aim  2:  Algorithms  for  Detection  of  Significant  Events 

After  the  completion  of  the  system  in  Aim  1,  the  research  team  designed  and  developed  and  conducted  preliminary  perception 
tests  using  the  f-EEG  system  to  collect  data  for  understanding  cognition  and  expertise  in  order  to  identify  significant  events.  For 
this  effort,  preliminary  tests  were  conducted  on  laboratory  personnel  to  assess  the  function,  accuracy  and  performance,  and 
usability  of  the  system  in  Aim  1  as  well  as  develop  a  protocol  for  deployment  of  the  system  for  experimental  studies  including 
software  set-up,  head-gear  calibration,  image  integration  in  the  eERS  system,  and  finally,  data  collection,  visualization,  and 
analyses. 

Once  tested,  the  wireless  f-EEG  experimentation  system  was  used  to  examine  the  existence  of  grandmother  cells.  For  these 
tests,  2  laboratory  test  subjects  provided  personalized  images  in  each  of  4  categories,  1-who  (i.e.  relative),  2-what  (i.e.  previous 
car  owned),  3-where  (i.e.  prior  vacation  location  or  significant  place  visited  previously),  and  4-significant  life  event  (i.e. 
graduation,  wedding,  etc).  These  images  were  then  combined  with  a  generic  database  containing  similar  images  of  people, 
things,  places,  and  events.  Each  test  subject  was  then  10  images  in  each  of  the  4  categories  with  their  own  image  randomly 
inserted  into  the  dataset  and  the  asked  to  click  a  mouse  upon  seeing  their  own  image.  Three  trials  were  repeated  for  each 
subject  and  category.  A  third  laboratory  person  (control)  was  also  shown  the  database  of  images  but  was  not  asked  to  provide 
any  personal  information. 

For  this  preliminary  test,  data  was  analyzed  by  comparing  the  control  to  the  two  test  subjects  to  see  if  EEG  response  to 
personalized  images  produced  appreciable  changes  as  compared  to  generic  images.  As  the  f-EEG  system  consisted  of  16 
spatially  distributed  EEG  nodes  (see  Fig.  6;  14  EEG  nodes  plus  2  ground  nodes,  not  shown),  the  research  team  was  able  to 
examine  spatio-temporal  changes  resulting  from  the  tests.  Data  for  each  of  the  16  EEG  nodes  were  grouped  according  to  head 
location  regions.  These  include  frontal  regions  (nodes:  AF3,  AF4,  F3,  F4,  F7,  F8,  FC5,  FC6),  back  regions  (nodes:  01, 02,  P7, 
P8,  T7,  T8),  and  sides  (front  left:  AF3,  F3,  F7,  FC5;  front  right:  AF4,  F4,  F8,  FC6;  back  left:  T7,  P7,  01;  back  right:  T8,  P8,  02). 
Thus,  taking  into  account  spatio-temporal  changes,  tracking  EEG  activity  by  region  overtime  we  created  spatial  distribution 


maps  of  activity  in  response  to  generic  vs.  personalized  images.  These  results  are  shown  in  Figure  7.  From  Fig.  7,  one  can 
see  the  redistribution  of  primary  brain  activity  spatially  over  time.  Initially,  when  viewing  generic  images,  brain  activity  center  in 
the  mid-brain  regions  and  moves  to  the  frontal  lobe  when  viewing  personalized  images,  and  returns  to  baseline  position  after 
these  images  are  removed. 

In  comparison  of  responses  between  each  of  the  different  categories  (who,  what,  where,  and  when),  analyses  from  the 
preliminary  trials  using  Kolmogorov-Smirnov  (KS)  plots  reveal  significant  differences  between  personalized  vs.  generic  image 
data  sets  (Figure  8a)  and  also  distinctly  different  responses  by  category,  with  the  strongest  association  to  recognition  of  people 
over  things,  places,  and  events  (Figure  8b). 

From  the  sample  data  collected,  algorithms  used  to  determine  recognition  of  significant  events  include  spatio-temporal 
transmission  metrics  related  to  brain  activity,  ratios  of  front-to-back  activity  and  front-to-side  activity  levels,  as  well  as  “strength” 
(i.e.  magnitude)  of  regional  brain  activity. 

C  Achievement  of  Aim  3:  Proposed  Schema  for  Fusion  of  Data  for  Training  of  Mixed  Expertise  Teams  and  Collective  Decision 
Support 

Preliminary  data  from  Aim  2  provide  evidence  of  grandmother  cells  that  respond  to  high  levels  of  expertise  and  training  and  are 
highly  specific  (i.e.  detection  of  true  positives).  By  measuring  spatio-temporal  dynamics  of  cognition  in  response  to  visual  stimuli 
(i.e.  environment,  situation  awareness),  it  is  expected  that  a  team  of  mixed  expertise  members  with  different  levels  of  training 
and  for  different  expertise  areas  will  respond  differently  to  critical  situations.  This  cognitive  response  can  be  measured  by  the  f- 
EEG  system  in  Aim  1.  Further,  quantitative  assessment  of  training  efficacy  (i.e.  increasing  expertise  in  subject  area)  may  be 
measured  as  shown  by  preliminary  evidence  shown  in  Aim  2. 

For  a  team  of  mixed  expertise  members  wearing  the  developed  f-EEG  system  (i.e.  multiple  headsets),  assessment  of  training 
can  be  performed  individually  and  objectively  by  examining  f-EEG  for  each  team  member  in  response  to  visual  stimulus  (i.e. 
hostile  scenarios,  critical  environments,  etc).  From  these  quantitative  assessment  scores,  a  composite  team  score  can  be 
derived  as  in  (1): 

Total  Score  (TS)=  (I_(i=1)An|  Em  j*  EwfH  _i  3  )/n  (1) 

where  m_i=member  training  score  (0-100%) 

Ewf3  J=member  weight  factor 
n=  number  of  team  members 

A  scenario  similar  to  the  test  situation  can  be  developed  where  images  are  shown  to  team  members  of  situations  that  are  safe 
vs.  hostile/critical  requiring  action.  Assessing  accuracy  of  these  responses  (by  f-EEG  and  mouseclick  or  keystroke)  combined 
with  time  duration  of  assessment  (dt)  results  in  a  derived  measure  of  throughput  rate  (TR).  As  training  level  increases,  it  is 
expected  that  throughput  rate  will  decrease  as  well  increasing  correctness. 

Testing  of  Aim  3  requires  human  subjects  and  IRB  approvals  that  are  beyond  the  scope  of  this  STIR  and  the  short  duration  of 
this  grant  period.  It  is  recommended  that  a  follow-up  project  be  initiated  using  the  developed  f-EEG  system  to  assess  collective 
team  training  of  mixed  expertise  members. 
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The  existence  of  grandmother  cells,  i.e.  hypothetical  cells  that  respond  to  specific  entities  with 
high  specificity,  have  long  been  theorized.  The  objective  of  this  project  was  to  apply  modem  technology 
and  know-how  to  develop  non-traditional  techniques  for  assessment  of  cognition  and  training 
effectiveness  of  mixed-expertise,  collaborative  teams.  This  effort  comprised  of  three  specific  aims:  1)  to 
develop  a  portable  wireless,  real-time  f-EEG  system  for  measurement  and  assessment  of  cognitive 
function  in  response  to  visual  stimuli;  2)  to  develop  algorithms  and  strategies  for  detecting  significant 
events  to  improve  situational  awareness;  and  3)  to  combine  aims  1  and  2  to  propose  a  strategy  for 
collective  decision  support  for  mixed  expertise,  collaborative  teams. 

Through  the  8  month  duration  of  this  STIR  proposal,  the  CUA  research  team  designed, 
developed  and  tested  the  function  of  a  portable,  real-time  f-EEG  system  (Aim  1)  that  is  able  to  implement 
and  deploy  cognitive  experimental  studies  of  visual  stimuli  in  various  forms  using  static  images  and/or 
video  content.  The  system  is  also  flexible  to  enable  cognitive  assessment  of  audio  content  as  well.  The 
system  uses  a  commercially  available  wireless,  16-node  EEG  headset  capable  of  collecting  continuous 
spatial  information  of  brain  function  in  response  to  external  stimuli.  The  headset  hardware  is  combined 
with  open  source,  easy-to-use,  drag-and-drop  software  developed  by  the  team  to  enable  i)  flexible  and 
customized  experimental  design,  implementation,  and  deployment,  ii)  data  collection  and  iii)  data 
visualization  and  analyses  as  proposed  in  Aim  1  of  the  project. 

Using  the  system  developed  in  Aim  1,  the  CUA  research  team  conducted  preliminary  trials  to 
develop  algorithms  and  strategies  to  detect  significant  events  in  response  to  visual  stimuli  and  to  study  the 
existence  of  grandmother  cells  (  Aim  2).  Preliminary  data  was  collected  to  compare  responses  to  visual 
stimulus  resulting  from  personalized  images  (i.e.  high  expertise)  versus  generic  images  from  a  standard 
dataset  (i.e.  low  expertise).  Preliminary  evidence  reveals  a  strong  and  differentiated  response  to 
personalized  images  compared  to  data  from  the  generic  image  set.  This  data  suggests  a  difference  in  the 
cognition  pathway  for  high  expertise  vs.  low  expertise  scenarios.  Algorithms  developed  to  assess 
cognition  response  include  analyses  of  temporal  responses  in  EEG,  spatial  transmission  dynamics  from 
occipital  (vision)-to-temporal  (memory)-to-frontal  (cognition)  lobes,  and  lastly  include  traditional  signal 
processing  metrics  such  as  power  spectral  density  (PSD).  Preliminary  results  show  that  strong  and 
significant  personal  identification  with  visual  stimuli  exhibit  unique  transmission  dynamics  and  response 
as  compared  to  viewing  generic  images.  This  provides  evidence  to  support  the  grandmother  neuron 
proposed  in  the  literature  and  may  have  high  potential  for  providing  quantitative  assessment  of  training 
effectiveness. 

Understanding  the  nature  of  the  changes  in  cognition  as  a  result  of  high  association  vs.  low 
association  with  visual  stimuli  may  provide  useful  quantitative  information  regarding  effectiveness  of 
training  as  it  relates  to  acquisition  of  expertise  and  learning.  The  system  in  Aim  1  has  the  potential  to 
provide  objective  measures  of  training  effectiveness.  Combining  the  system  achieved  in  Aim  1  with  the 
preliminary  evidence  from  Aim  2,  a  multi-user  f-EEG  system  as  applied  to  a  mixed  expertise, 
collaborative  team  has  the  potential  for  quantitative  assessment  of  individual  and  collective  team  training 
performance.  This  performance  measure  can  serve  as  a  measure  of  accuracy  (i.e.  correctness)  and 
combined  with  time  duration  of  training,  can  be  used  for  measuring  throughput  rate  for  individuals  as 
well  as  collectively  for  the  team. 
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I.  Project  Objective: 

This  proposal  seeks  to  apply  modern  technology  and  know-how  to  develop  a  non- 
traditional  methodology  of  image  processing  of  salient  feature  extraction  to  information 
processing  and  fusion  in  tenns  of  human  visual  system  (HVS)  in  order  to  train  a  mix  of  human 
interdisciplinary  experts  to  increase  the  throughput  rate  of  collective  decision  making.  We 
define  the  decision  throughput  rate  as  the  time  duration  multiplied  by  the  degree  of  correctness . 
This  research  proposes  development  and  refinement  of  a  portable,  wireless,  real-time,  and 
smartphone-based  /i//?c//w?<;//-elcctroencephalogram(/-EEG)  system  using  high-density  dry 
electrodes  and  compressive  sensing  strategies  to  address  spatial  and  temporal  sparseness 
limitations.  Further,  the  R&D  effort  seeks  to  refine  algorithms  for  information  processing, 
feature  extraction,  and  fusion  algorithms  for  assessment  of  military  training,  assessment  of 
situational  awareness,  and  for  collective  decision  support.  The  overall  impact  of  the  proposed 
work  is  the  development  of  a  processing  algorithm  interface  tool  for  adaptive  real-time  feedback 
of  knowledge  acquisition  correlating  to  long-term  (i.e.  hippocampus)  with  short-term  (i.e.  frontal 
lobe)  working  memory  for  support  of  complex,  collective  team  decision-making. 

Specifically,  the  proposed  effort  advances  prior  work  by  our  team  and  will  develop 
necessarily  software  to  extract  multiple  picture  (e.g.  Grandmother  image  feature)  in  a  sparse 
coding  for  pseudo-orthogonal  span  of  the  search  space  and  model  hetero-associative  memory  at 
Hippocampus  to  enhance  grandmother  recall  in  other  confused  SNR.  This  work  has  three 
specific  aims: 

1)  conduct  basic  research  and  assessment  of  a  portable,  wireless,  real-time  f-EEG  system  for 
measurement  of  grandmother  cells  (i.e.  selective  visual  neurons  with  high  specificity  for 
targeted  visual  objects)  located  in  the  human  hippocampus, 

2)  development  of  algorithms  and  strategies  for  detection  of  significant  events  using 
artificial  neural  networks  approaches  to  improve  situational  awareness,  and 

3)  fusion  of  aim  1  (f-EEG  for  cognition)  and  aim  2  (detection  of  significant  events)  as 
applied  to  mixed  expertise  teams  for  collective  decision  support. 

The  outcome  of  the  proposed  research  effort  will  result  in  an  improved  real-time,  wireless, 
portable  f-EEG  system  capable  of  collecting  neuro-information  for  processing  and  fusion  of  data 
in  order  to  detect  significant  events  and  to  improve  situational  awareness  and  for  collective  team 
decision  support  in  hostile  and/or  critical  environments. 

II,  Approach  &  Scientific  Accomplishments 

Section  II  herein  describes  the  scientific  accomplishments  for  the  project  in  relation  to  each  of  the 
proposed  efforts  of  the  STIR  proposal. 


II.  A.  Achievement  of  Aim  1:  Wireless,  Real-time  f-EEG  Experimentation  System 

Since  the  start  of  this  STIR  contract  (  August  2014),  the  focus  of  the  effort  has  mainly  been  on 
development,  refinement,  and  integration  of  the  portable,  wireless,  real-time  f-EEG  system  proposed  in 
Aim  #1.  The  system  uses  a  commercial  off-the-shelf  (COTS)  16-node,  dry  electrodes  EEG  head  mounted 
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device  (Emotiv  Inc.,  Figure  1)  for  collection  of  EEG  data.  This  device  is  wireless  and  transmits  data  via 
Bluetooth  to  a  PC  using  a  USB  dongle.  The  advantage  of  the  system  over  others  is  the  ability  to  collect 

high  resolution  EEG  data  without  complicated  setup  and  calibration 
as  with  EEG  nets  and  similar  devices.  For  the  end  user,  donning  the 
device  is  similar  to  wearing  and  positioning  of  headphones.  Further, 
data  output  and  sensitivity  is  minimally  affected  by  hair  and  other 
artefact. 

The  EEG  headset  integrates  with  a  custom  designed  EEG 
experimental  management,  data  collection  and  analysis  platform 
comprised  of  both  hardware  and  software  components.  At  the  heart 
of  the  system  is  the  EEG  Experiment  Runtime  Service  (eERS) 
which  supports  experimental  set-up,  experimental  design  and 
configuration,  set-up  for  incoiporation  of  visual  stimuli, 
management  of  incoming  data,  and  data  analytics.  eERS  is 
developed  with  an  open  source  strategy  and  future  deployment  is 
expected  to  be  cloud-based  to  allow  for  scalability,  portability  and 
versatility.  Figure  2  shows  the  EEG  experimental  platform 
architecture. 

The  eERS  system  allows  for  single  or  multi-user  EEG 
analyses.  For  multi-user  analyses,  each  user  focuses  on  his/her  own  screen  (i.e.  viewpoint)  and  data  is 
collected  using  the  personalized  headset.  This  EEG  data  is  managed  and  synchronized  with  other  users,  as 
appropriate,  through  the  collection  proxy  and  is  also  synchronized  with  events  in  visual  stimuli.  Care  in 
development  has  been  taken  to  address  issues  of  data  latency  resulting  from  incoming  transmission  from 
remote  users.  Additionally,  missing  packets  resulting  are  also  handled  by  the  collection  proxy.  Once  data 


Figure  1:  Simple,  wireless  Emotiv 
neuro -headset  replaces  complicated 
EEG  nets  for  EEG  data  collection. 


EEG  Data  Collection/Management  Platform  Architecture 


Figure  2:  Architecture  for  EEG  data  collection  and  management  platform.  Experiment  Runtime  Service  (ERS) 
module  controls  experimental  setup  and  configuration,  connection  to  webservices,  data  collection  proxy,  and  data 
analytics.  Experiments  can  be  run  with  one  user  (i.e.  individual)  or  with  a  group  of  users  (i.e.  collective).  For  multi¬ 
user  experimentation,  users  can  all  be  local  or  remotely  connected  via  web-GUI  interface. 
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arrives,  it  is  processed  by  the  data  analytics  engine  which  integrates  open-source  analytics  engine  by 
Octave.  Algorithms  are  developed  for  key  EEG  feature  extraction.  EEG  data  from  each  of  the  16  headset 
nodes  is  combined  to  provide  3D  mapping  of  activated  brain  regions  responding  to  stimuli.  Figure  3 
provides  a  sample  of  raw  EEG  collected  from  the  wireless,  EEG  headset  system  along  with  a  fused 
mapping  in  3D  of  activated  regions  responding  to  stimuli. 

An  important  aspect  of  the  development  of  the  eERS  system  includes  the  development  of  the 
experimental  set-up  engine  (Figure  4).  This  portion  of  the  software  development  allows  for  customized 
experimental  set-up  by  trained  as  well  as  minimally  trained  users.  It  utilizes  an  icon-driven,  drag-n-drop 
experimental  setup  schema  where  key  components  required  for  the  experiment  (i.e.  single  vs.  multi  user, 
visual  stimuli,  data  outputs,  etc)  can  be  easily  designed  and  implemented.  Various  stimuli  (i.e.  images, 
video,  audio)  as  well  as  input  devices  (i.e.  keyboard,  mouseclicks)  can  be  integrated  and  collected  by  the 
eERS  system. 


Drag-n-Drop  Set-Up  & 

Device  Configuration 


[  Use' J  fe  ^ 

j  User  n 
Connect  to  Host 

▼ 


Web 

Services 


Set-up  stimulus  scenario 


EEG  Experimental 
Runtime  Service 


Figure  3:  Output  from  eERS  system  showing  raw  16- 
node  EEG  data  (left)  and  fused,  3D  mapping  of 
activated  brain  regions  responding  to  stimuli  (right). 


Figure  4:  EEG  Experimental  set-up  engine  facilitates 
customization  of  experiments  through  drag-n-drop 
interface. 


II.B  Achievement  of  Aim  2:  Algorithms  for  Detection  of 
Significant  Events 

After  the  completion  of  the  system  in  Aim  1,  the  research 
team  designed  and  developed  and  conducted  preliminary 
perception  tests  using  the  f-EEG  system  to  collect  data  for 
understanding  cognition  and  expertise  in  order  to  identify 
significant  events.  For  this  effort,  preliminary  tests  were 
conducted  on  laboratory  personnel  to  assess  the  function, 
accuracy  and  performance,  and  usability  of  the  system  in  Aim  1 
as  well  as  develop  a  protocol  for  deployment  of  the  system  for 


Figure  5:  Fleadset  set-up  and  calibration. 
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experimental  studies  including  software  set-up,  head-gear  calibration,  image  integration  in  the  eERS 
system,  and  finally,  data  collection,  visualization,  and  analyses. 

Once  tested,  the  wireless  f-EEG  experimentation  system  was  used  to  examine  the  existence  of 
grandmother  cells.  For  these  tests,  2  laboratory  test  subjects  provided  personalized  images  in  each  of  4 
categories,  1-who  (i.e.  relative),  2-what  (i.e.  previous  car  owned),  3-where  (i.e.  prior  vacation  location  or 
significant  place  visited  previously),  and  4-significant  life  event  (i.e.  graduation,  wedding,  etc).  These 
images  were  then  combined  with  a  generic  database  containing  similar  images  of  people,  things,  places, 
and  events.  Each  test  subject  was  then  10  images  in  each  of  the  4  categories  with  their  own  image 
randomly  inserted  into  the  dataset  and  the  asked  to  click  a  mouse  upon  seeing  their  own  image.  Three 
trials  were  repeated  for  each  subject  and  category.  A  third  laboratory  person  (control)  was  also  shown  the 
database  of  images  but  was  not  asked  to  provide  any  personal  information. 

For  this  preliminary  test,  data  was  analyzed  by  comparing 
the  control  to  the  two  test  subjects  to  see  if  EEG  response  to 
personalized  images  produced  appreciable  changes  as 
compared  to  generic  images.  As  the  f-EEG  system  consisted 
of  16  spatially  distributed  EEG  nodes  (see  Fig.  6;  14  EEG 
nodes  plus  2  ground  nodes,  not  shown),  the  research  team 
was  able  to  examine  spatio-temporal  changes  resulting  from 
the  tests.  Data  for  each  of  the  16  EEG  nodes  were  grouped 
according  to  head  location  regions.  These  include  frontal 
regions  (nodes:  AF3,  AF4,  F3,  F4,  F7,  F8,  FC5,  FC6),  back 
regions  (nodes:  01,  02,  P7,  P8,  T7,  T8),  and  sides  (front  left: 

AF3,  F3,  F7,  FC5;  front  right:  AF4,  F4,  F8,  FC6;  back  left:  Figure  6:  Emotiv  EEG  node  locations. 

T7,  P7,  01;  back  right:  T8,  P8,  02).  Thus,  taking  into 

account  spatio-temporal  changes,  tracking  EEG  activity  by  region  over  time  we  created  spatial 
distribution  maps  of  activity  in  response  to  generic  vs.  personalized  images.  These  results  are  shown  in 
Figure  7.  From  Fig.  7,  one  can  see  the  redistribution  of  primary  brain  activity  spatially  over  time. 

Initially,  when  viewing  generic  images,  brain  activity  center  in  the  mid-brain  regions  and  moves  to  the 
frontal  lobe  when  viewing  personalized  images,  and  returns  to  baseline  position  after  these  images  are 
removed. 


(a)  (b)  (c) 

Figure  7:  3-D  visualization  of  brain  activity  for  test  subject  2  (a)  before,  (b)  during,  and  (c)  after  being 
shown  personalized  images. 
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In  comparison  of  responses  between  each  of  the  different  categories  (who,  what,  where,  and  when), 
analyses  from  the  preliminary  trials  using  Kolmogorov- Smirnov  (KS)  plots  reveal  significant  differences 
between  personalized  vs.  generic  image  data  sets  (Figure  8a)  and  also  distinctly  different  responses  by 
category,  with  the  strongest  association  to  recognition  of  people  over  things,  places,  and  events  (Figure 
8b). 


(a)  (b) 


Figure  8:  KS  plot  comparing  response  (a)  before  (blue)  and  after  (red)  shown  personalized  image  and 
(b)  comparing  response  to  “who  ”  vs.  “where  ”  images. 

From  the  sample  data  collected,  algorithms  used  to  determine  recognition  of  significant  events 
include  spatio-temporal  transmission  metrics  related  to  brain  activity,  ratios  of  front-to-back  activity  and 
front-to-side  activity  levels,  as  well  as  “strength”  (i.e.  magnitude)  of  regional  brain  activity. 


II.  C  Achievement  of  Aim  3:  Proposed  Schema  for  Fusion  of  Data  for  Training  of  Mixed  Expertise 

Teams  and  Collective  Decision  Support 

Preliminary  data  from  Aim  2  provide  evidence  of  grandmother  cells  that  respond  to  high  levels  of 
expertise  and  training  and  are  highly  specific  (i.e.  detection  of  true  positives).  By  measuring  spatio- 
temporal  dynamics  of  cognition  in  response  to  visual  stimuli  (i.e.  environment,  situation  awareness),  it  is 
expected  that  a  team  of  mixed  expertise  members  with  different  levels  of  training  and  for  different 
expertise  areas  will  respond  differently  to  critical  situations.  This  cognitive  response  can  be  measured  by 
the  f-EEG  system  in  Aim  1.  Further,  quantitative  assessment  of  training  efficacy  (i.e.  increasing  expertise 
in  subject  area)  may  be  measured  as  shown  by  preliminary  evidence  shown  in  Aim  2. 

For  a  team  of  mixed  expertise  members  wearing  the  developed  f-EEG  system  (i.e.  multiple 
headsets),  assessment  of  training  can  be  performed  individually  and  objectively  by  examining  f-EEG  for 
each  team  member  in  response  to  visual  stimulus  (i.e.  hostile  scenarios,  critical  environments,  etc).  From 
these  quantitative  assessment  scores,  a  composite  team  score  can  be  derived  as  in  (1): 

Total  Score  ( TS )  =  ^=^*wfi  (1) 

where  mt  =  member  training  score  (0  —  100%) 
wf  =  member  weight  factor 
n=  number  of  team  members 
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A  scenario  similar  to  the  test  situation  can  be  developed  where  images  are  shown  to  team  members 
of  situations  that  are  safe  vs.  hostile/critical  requiring  action.  Assessing  accuracy  of  these  responses  (by  f- 
EEG  and  mouseclick  or  keystroke)  combined  with  time  duration  of  assessment  (dt)  results  in  a  derived 
measure  of  throughput  rate  (TR).  As  training  level  increases,  it  is  expected  that  throughput  rate  will 
decrease  as  well  increasing  correctness. 

Testing  of  Aim  3  requires  human  subjects  and  IRB  approvals  that  are  beyond  the  scope  of  this  STIR 
and  the  short  duration  of  this  grant  period.  It  is  recommended  that  a  follow-up  project  be  initiated  using 
the  developed  f-EEG  system  to  assess  collective  team  training  of  mixed  expertise  members. 


III.  Scientific  Barriers 

The  development  of  the  f-EEG  system  in  Aim  1  progressed  as  planned.  While  operating  wireless,  the 
EEG  headset  currently  requires  close  proximity  (-10-15  feet)  to  the  computer  acquisition  system  due  to 
bandwidth  requirements.  This  constraint  limits  potential  scenarios  of  ambulatory  acquisition  of  f-EEG 
data  (i.e.  in  the  field  training  assessment).  In  some  cases  where  test  subjects  have  excessive  amounts  of 
head  hair  (i.e.  females),  the  headset  EEG  nodes  do  not  have  good  contact  and  acquisition  may  be  dropped 
in  some  nodes  requiring  timely  repositioning  and  adjustment. 

While  test  data  was  collected  on  the  investigators  for  Aim  2,  there  was  insufficient  time  for  the  short 
duration  of  this  project  to  obtain  IRB  from  ARO  and  CUA  to  conduct  trials  in  a  larger  sample  size.  This  is 
the  next  step  in  the  progression  of  this  project.  The  pilot  data  was  sufficient  for  the  team  to  develop  data 
visualization,  analyses  and  fusion  strategies  to  assess  levels  of  cognition  from  visual  stimuli. 


IV.  Scientific  Significance 

The  scientific  significance  of  this  research  effort  is  first  the  development  of  a  mobile,  wearable, 
wireless  f-EEG  system  for  collection  of  functional  EEG  data.  Such  a  system  here-to-fore  has  not  yet  been 
realized.  While  the  proposed  application  here  is  for  studying  adaptive  training  and  mixed  expertise, 
collective  decision-making,  the  prototype  EEG  experimental  runtime  service  platform  has  broader 
significance  in  brain  research.  It  has  the  potential  to  be  used  to  study  mental  health  disorders  and 
traumatic  brain  injury.  Potential  applications  directly  related  to  DOD-Army  for  assessment  of  PTSD  for 
returning  soldiers  as  well  as  dementia  and  Alzheimer’s  in  older  veterans.  Current  tools  for  assessing  brain 
function  have  various  drawbacks  that  make  them  inapplicable  for  the  intended  application. 

Further,  preliminary  evidence  from  pilot  data  collected  as  part  of  Aim  2  shows  evidence  of  a 
grandmother  cells  that  are  highly  specific  to  personalized  images.  This  response  is  more  pronounced  in 
facial  recognition  of  people  than  in  places,  things,  or  events.  This  merits  further  study. 

V.  Collaborations  and  Leveraged  Funding 

We  are  currently  collaborating  with  Dr.  Harold  Szu  at  Army’s  Night  Vision  and  Electronic  Sensors 
Directorate  (NVESD)  on  the  current  project.  Further,  our  team  has  active  collaborations  with  NIH’s 
National  Institute  on  Mental  Health  (NIMH;  Drs.  Lalonde  and  Gotay)  as  part  of  the  Brain  Order  Disorder 
(BOD)  Working  Group. 
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VI.  Technology  Transfer 

In  addition  to  the  scientific  merits  of  this  work,  we  believe  there  are  several  opportunities  for 
technology  transfer  related  to  the  project.  While  currently,  the  project  is  in  its  nascent  stage,  we  believe 
there  are  opportunities  for  the  eERS  system  to  be  directly  useful  for  brain  researchers  and  clinicians  as  it 
provides  an  opportunity  for  quantitative  assessment  of  brain  function.  The  system  may  be  useful  in 
assessing  PTSD,  dementia,  concussions,  etc.  as  well  as  effectiveness  of  treatment  programs.  Another 
potential  application  is  for  the  use  of  the  system  for  cognitive  assessment  at-home  individuals.  We  expect 
this  system  will  also  be  useful  in  assessing  cognition,  learning,  and  collective  decision  making  that  may 
be  universally  useful  for  educators.  Further,  there  is  an  opportunity  for  the  analytics  engine  to  be  useful 
for  analyses  of  laboratory  collected  EEGs  as  a  service. 

We  will  work  with  CUA’s  technology  transfer  office  as  the  project  progresses  related  to  intellectual 
property  and  licensing  opportunities. 

VII.  Future  Research  Plans 

Future  research  plans  related  to  this  project  include  human  subjects  studies  to  achieve  the  3  aims  of 
the  project.  Once  IRB  approval  is  obtained,  we  will  be  collecting  studies  using  the  developed  eERS 
system  to  measurement  of  grandmother  cells  and  refining  algorithms  for  detection  of  significant  events. 
We  have  submitted  a  research  proposal  to  CUA’s  IRB  to  expand  the  efforts  of  Aim  2.  We  intend  to 
submit  for  follow-up  funding  to  ARO  and  to  NIH  for  expansion  of  data  collection  related  to  Aim  2  (NIH 
for  assessing  cognitive  function  in  acute  and  chronic  mental  health  conditions)  and  Aim  3  (  ARO  for 
collective  training  and  decision  support)  of  this  project. 

Beyond  the  duration  of  the  project,  we  intend  to  pursue  further  funding  (sources  include  ARO,  NSF, 
NIH)  related  to  data  fusion  and  3-D  visualization  of  complex  EEG  data  collected.  Another  challenge  will 
be  to  reduce  the  number  of  EEG  nodes  required  for  sensitive  and  high  resolution  EEG  data.  Our  goal  is  to 
reduce  the  16-node  system  to  one  requiring  between  3-5  nodes.  This  effort  would  simplify  experimental 
set-up  and  calibration,  especially  important  as  one  deploys  the  technology  out  in  the  field  and  is  being 
used  by  less  trained  individuals. 

VIII.  Anticipated  Scientific  Accomplishments 

The  outcome  of  the  research  effort  has  resulted  in  an  integrated  real-time,  wireless,  portable 
f-EEG  system  capable  of  collecting  neuro-infonnation  for  processing  and  fusion  of  data  in  order 
to  detect  significant  events  and  to  improve  situational  awareness  and  for  collective  team  decision 
support  in  hostile  and/or  critical  environments.  The  system  will  include  both  hard  and  software 
components  as  well  as  an  advanced  analytics  engine  for  quantitative  assessment. 
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